Statistical Strategies for Pruning All the Uninteresting Association Rules

We propose a general framework to formalize the problem of capturing the intensity of implication for association rules through statistical metrics. In this framework we present properties that influence the interestingness of a rule, analyze the conditions that lead a measure to perform a perfect prune at a time, and define a final proper order to sort the surviving rules. We will discuss why none of the currently employed measures can capture objective interestingness, and just the combination of some of them in a multi-step fashion, can be reliable. In contrast, we propose a new simple modification of the Pearson coefficient that will meet all the necessary requirements. We statistically infer the convenient cut-off threshold for this new metric by empirically describing its distribution function through simulation. Experiments show a promising behaviour of our proposal. 1 PROBLEM FORMULATION One of the most relevant tasks in Knowledge Discovery in Databases is mining for association rules in large masses of data, as it was first formulated by [1]. This task is often decomposed into two separate phases: 1/ Finding all the frequent itemsets having support over a user-specified threshold, and, 2/ Generating the association rules from the maximal discovered frequent itemsets. The input of a frequent sets algorithm is a database , composed of a collection of transactions, where each transaction is a subset of a given fixed set of items . Let be an itemset, and let ! "# $ &% be the ratio of the number of transactions in which appears to the number of all transactions in , i.e. ! "# $ '%( *) +-, . /-02143 576 8 5 8 . We note the support of an itemset as ! "# $ '% . An itemset is called frequent if its support exceeds a given user-specified threshold, 9 . In the second phase, association rules are constructed from those maximal frequent sets. In brief, given any maximal frequent itemset : , an association rule is an expression